Information is frequently recorded and communicated with pen and paper. Such paper-bound information, however, is difficult to manage and communicate efficiently. Computers are used to an increasing extent for communicating and managing information. Such computer-based information is typically inputted via a keyboard and stored in the memory of the computer, for instance on a hard disk. The inputting of information via a keyboard, however, is slow and typing errors are frequent. Although many utility programs exist, such as word division programs, to facilitate computer inputting, it is not very convenient to read large amounts of text on a display. When information is available in the computer, however, it can easily be communicated to others, such as by e-mail or SMS (short message service) via an Internet connection or as a fax via a fax modem.
Patent Application PCT/SE00/01895, filed by the applicant, discloses a system in which a pen and a sheet of paper are used to take down information in the traditional manner, while at the same time a digital graph is formed of several traces or strokes by movement of the pen over the sheet of paper, such that the graph can be transferred to a computer. Such a system combines the advantage of handling pen and paper, which many users are accustomed to, with the computer's superior capability of communicating and storing information. The paper is provided with a coding pattern, for instance consisting of points or marks. The pen has an optical sensor which records the coding pattern and, with a mathematical algorithm, calculates the position of the pen on the coding pattern.
The traditional pen will in this way be an excellent input device for the computer, and the computer can be used to store the information instead of filing sheets of paper. Moreover, the information can easily be communicated by means of the computer. The disadvantage of this method of inputting is that the information is graphical and not in character format, to which the computer is adapted.
However, the information included in the graph contains additional information which can be used for various purposes. Specifically:
1) The information comprises an image, such as figures or lines that are related to each other and which can be interpreted by a human being, such as letters, a symbol, a figure or drawing. This is the actual message which is taken down and which the user wants to manage in one way or another, for instance file the message or send it to a recipient. This information is referred to herein as message information and is stored in a graphical format, for instance in a vector format or as a collection of pixels.
2) The part of the message information which consists of letters (handwritten) can be subjected to subsequent processing in the form of optical character recognition (OCR) or intelligent character recognition for conversion into a character format which can be used by the computer, for instance for search purposes or cataloguing. Also, symbols can be interpreted, for instance shorthand symbols or icons, which the user predefines to have a specific meaning. This information is referred to herein as character information.
3) The information may further comprise an identification of the pen which is used to take down the information. The identification of the pen is referred to herein as the pen ID.
4) Finally, the graph contains information about where on the surface the graph was taken down, so-called absolute position information. As described in detail below, the present invention relates to a number of services or applications based on the absolute position information.
There are previously known systems for obtaining absolute position information (see for instance U.S. Pat. No. 5,852,434). These previously known systems, however, only disclose the use of such information to form message information, i.e., group 1) above.
The information can be processed at different points in the system. The pen thus comprises: an image sensor; a processor; associated memory; and a battery. Moreover, the pen comprises a communication unit, such as a cord for connecting to a computer or an IR link or short range radio link for communication with the computer. The pen also includes a pressure sensor which records whether the pen point is in touch with a base. The pressure sensor can also detect the pressure by which the pen point is pressed against the base.
Even if it is possible to transfer the information from the pen to an external computer in the form of the video image which is reproduced by the sensor and all computer processing then takes place in the computer, it is preferred that the pen be provided with a processor and some image processing capability. Thus, the pen includes a processor or logic which processes the obtained video image and calculates the positions of the marks on the sensor's reproduction of the surface of the sheet of paper. This image processing includes adjusting for perspective effects caused by rotation and inclination of the pen relative to the paper, and compensating for different light conditions. Preferably, the pen also comprises a computer program which, with the aid of algorithms, calculates the x-y coordinates of the pen. It is thus made possible for the pen to perform different functions based on the values of the coordinates, as will be explained in more detail below.
The information is thus stored first in the memory of the pen, in the form of a sequence of coordinates which can easily be converted into vectors. This message information in vector form can be transferred to a computer which executes a program to output a graph on the computer display. The message information can also be sent to a printer to be printed immediately or to a dumb display which only has the function of drawing the graph on a display, such as a TV screen. Interpretation of the message information into character format can take place either in the pen or in the computer.
It may be desirable to communicate message information to a recipient, for instance in the form of a fax or an e-mail. The pen can cooperate with a mobile phone, by means of a cord, IR radiation or a radio link. The mobile phone essentially functions as a “modem” to link the message information to a fax number, or to the Internet by calling a modem pool of an Internet operator. In this case, it is in some cases necessary for part of the message information to be subjected to character recognition, preferably in the pen, and to be used as a telephone number or an IP address for communication via the Internet. Alternatively, an address function in the mobile phone can be used.
Of course, the computer may act as a “modem” as well and in that case character recognition can take place in the computer, which may often have better processor capacity than what is provided in the pen. In certain cases, communication via the Internet can first take place to a server at an IP address preprogrammed in the pen (or the mobile phone), where character recognition can take place.